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IMAGE CONVERSION AND ENCODING TECHNIQUES 

FIELD OF INVENTION 

The present invention is generally directed towards a technique for 
converting 2D images, that have been formed from a layered source, into 3D. It 
5 is also an objective of the invention to enable the resulting images to be stored 
and transmitted in a 2D compatible format and the images converted into 3D at 
the viewing location either prior to viewing or in real time. 
BACKGROUND 

The applicants have previously described in PCT/AU96/00820, a method of 
10 producing left and right eye images for a stereoscopic display from an original 2D 
image including the steps of: 

a. identifying at least one object within the original image; 

b. outlining each object; 

c. defining a depth characteristic for each object; and 

15 d. respectively displacing selected areas of each object by a determined 
amount in a lateral direction as a function of the depth characteristic of each 
object, to form two stretched images for viewing by the left and right eyes of the 
viewer. 

These steps can be individually and collectively referred to as Dynamic Depth 
20 Cueing or DDC. 

Additionally, the Applicants have previously described in PCT/AU98/01 55 in 
one aspect a method of encoding a depth map including the steps of 

a. allocating an object number to an object 

b. allocating the object with a depth; and 
25 c. defining the object outline 

The object outline may be defined by a series of co-ordinates, and/or curves. 

In another aspect the previous invention disclosed the use of curves to 
generate an outline of an object in a 2D to 3D conversion process. 

In a further aspect the previous invention disclosed the use of curves to define 
30 an object in a 2D to 3D conversion process. 

In a further aspect the previous invention disclosed a method of transmission 
of depth map information wherein the information is included in the Vertical 
Blanking Interval or MPEG data stream. 




In a further aspect the previous invention disclosed the use of generic libraries 
to assist in the 2D to 3D conversion process. 

Additionally, the applicants have previously described in PCT/AUOO/00700 a 
method of encoding a depth map including: 
5 a. identifying and outlining an object in an image; 

b. allocating an object identification symbol to an object; 

c. using the allocated object symbol to represent the shape of the object; 

d. allocating the object with a depth; 

e. compressing the information representing the object and its depth; 
10 f. transmitting and/ or storing this information; and 

g. decompressing the information. 

Additionally this application disclosed a method of encoding a depth map 
including: 

a. Identifying and outlining an object within an image; 
15 b. allocating an object identification symbol to the object; 

c. defining the object by drawing a plurality of lines across the image, and 
determining the start and finish positions of each said line, wherein a new line is 
commenced each time an object boundary is reached; and 

d. allocating the object with a depth. 

20 In PCT/AU96/00820 a number of techniques for producing DDC data for 

live broadcasting were disclosed. These included, although were not limited to; 

Determining the distance of objects from a camera using variable focus 
techniques; and 

jhe use of two cameras and an autocorrelator to determine the distance of 
25 objects from a camera. 

In effect, these techniques described what has now become generally 
referred to as the production of a depth map, where each object in a scene is 
allocated a unique attribute (usually a shade of grey) depending upon the 
relative, or absolute, distance from the object to a reference point e.g. the camera 
30 lens. 
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The Applicants have previously disclosed how such depth maps may be 
compressed and included in 2D images to enable these DDC encoded image to 
be subsequently converted to a stereoscopic image. 

Additionally, the applicants have previously described in PQ4166 a method 
5 that takes an existing depth map and represents it in terms of curves and ramp 
functions including: 

a. Applying a curve to the outline of an object within the depth map; 

b. Representing the continuous depth value within the outline with a ramp 
function; and 

10 c. Encoding and compressing the data representing the cun/e and depth 
function 

What has not been previously disclosed is a process that takes an existing 2D 
image that is comprised of a layered source, allocates a depth to each object on 
each layer and convert the image into 3D. 
1 5 OBJECTIVE OF THE INVENTION 

It is an objective of the present invention to provide a relatively simple 
technique for converting 2D images, that have been formed from a layered 
source, into 3D. It is also an objective of the invention to enable the resulting 
images to be stored and transmitted in a 2D compatible format and the images 
20 converted into 3D at the viewing location, either prior to viewing or in real time. It 
is proposed that 2D compatibility will be provided by applying the previously 
disclosed DDC techniques. 
SUMMARY OF THE INVENTION 

With the above objective in mind, the present invention provides in one 
25 aspect a method of converting 2D images from a layered source into 3D for a 
stereoscopic display requiring a left and right eye image, or a 2D image and an 
associated depth map or multiple images and includes the steps of: 

defining a depth characteristic for each object on each layer, and 
respectively displacing each object on each layer by a determined amount 
30 in a lateral direction as a function of the depth characteristic of each iayer; and 

including the information defining the depth characteristic for each object 
on each layer within the original 2D image such that the image may be viewed in 



its original 2D form or subsequently converted into a fornnat suitable for viewing 
stereoscopically. 

BRIEF DESCRIPTION OF THE DRAWINGS 

To provide a better understanding of the present invention, reference is 
5 nnade to the acconnpanying drawings which illustrate a preferred embodinnent of 
the present invention. 
IN THE DRAWINGS 

Figure 1 shows a layered 2D innage. 

Figure 2 shows how the composite image in Figure 1 is composed of 
1 0 objects existing on separate layers. 

Figure 3 shows how left and right eye images are formed. 
DETAILED DESCRIPTION OF THE INVENTION 

In the preferred embodiment, the conversion technique includes the 
following steps: 

15 IDENTIFY EACH OBJECT ON EACH LAYER AND ASSIGN A DEPTH 

CHARACTERISTIC TO EACH OBJECT 

The process to be described is intended to be applied to 2D images that 

are derived from a layered source. Such images include, but are not limited to, 

cartoons, MPEG video sequences (in particular video images processed using 
20 MPEG4 where each object has been assigned a Video Object Plane) and 

Multimedia images intended for transmission via the Internet, for example images 

presented in Macromedia "Flash" format. 

Such a layered 2D image is shown in Figure 1 . Figure 2 illustrates how the 

composite image in Figure 1 is composed of objects existing on separate layers 
25 and consolidated so as to form a single image. It will be appreciated by those 

skilled in the art that the separate layers forming the composite image may also 

be represented in a digital or video format. In particular it should be noted that 

the objects on such layers may be represented in a vector format. 

Where more that one object is present on a specific layer it may be 
30 desirable to further segment the objects into additional layers to enhance the 3D 

effect. 
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In the preferred embodiment, each layer, and object within the layer, is 
assigned an identifier. In addition, each object is assigned a depth characteristic 
in the manner previously disclosed in application PCT/AU98/0155 that is hereby 
included by reference. 
5 The general format of the object definition is therefore: 

<layer identifierxobject identifierxdepth characteristio 
where the identifier can be any alpha numeric identifier and the depth 
characteristic is as previously disclosed. 

It is an intention of this invention to disclose the addition of a depth 
10 characteristic identifier to existing layer based image storage and transmission 
protocols that may already identify objects within an image by other means. 

In the simplest implementation the layer identifier may be used as a direct, 
or referred, reference to the object depth. 

For example purposes only, consider a 2D image comprising 4 layers with 
15 each layer containing a single object. The layers may be numbered 1 to 4 and 
ordered such that, when displayed stereoscopically, the object on layer 1 appears 
closest to the viewer, the object on layer 2 appears behind the object on layer 1 
etc such that the object on layer 4 appears furthest from the viewer. It will be 
obvious to those skilled in the art that this sequence could be reversed i.e. layer 4 
20 could contain an object that is closer to the viewer and layer 1 an object furthest 
from the viewer or a non sequential depth or non linear representations applied. 

This simple implementation has the disadvantage that should additional 
layers be introduced or removed during the 2D sequence then the overall depth 
of the image may vary between scenes. The general form of the object definition 
25 overcomes this limitation by separating the identifiers relating to object depth and 
layer. 

LATERALLY DISPLACE EACH LAYER 

It has been disclosed in the prior art that a stereoscopic image can be 
created by copying an object in a 2D image, laterally displacing it, and replacing 
30 the displaced object in a copy of the original 2D image so as to form a stereo 
pair. The prior art teaches that the amount of lateral displacement, and the 
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relative direction, determines how far away from the viewer the object appears 
when viewed stereoscopically. 

It has also been disclosed in the prior art how a stereo pair can be created 
by cutting an object out of a 2D image, laterally shifting it and pasting it back into 
5 a copy of the original image. This creates a hole in the original 2D image which is 
undesirable. The applicant has previously disclosed in PCT/AU96/00820 an 
alternative method of creating a stereoscopic image from a 2D source by 
stretching objects in the 2D image that overcomes the limitations of the prior art. 
For purpose of explanation only it is assumed that the 2D image_is 

10 composed of a number of objects that exist on separate layers. It is also 
assumed that the 2D image is to be converted to 3D and displayed on a 
stereoscopic display that requires separate left and right eye images. The layers 
are sequenced such that the object on layer 1 is required to be seen closest to 
the viewer when converted into a stereoscopic image and the object on layer n 

1 5 furthest from the viewer. 

Also, for purpose of explanation only, it is also assumed that the object 
depth is equal to, or a function of, the layer number. It is also assumed that the 
nearest object i.e. layer 1, will have zero parallax on the stereoscopic viewing 
device and that all other objects on sequential layers will appear behind 

20 successive objects. 

In order to produce the left eye image sequence a copy of layer 1 of the 
2D image is made. A copy of layer 2 is then made and placed below layer 1 with 
a lateral shift to the left. The amount of lateral shift is determined so as to 
produce an aesthetically pleasing stereoscopic effect or in compliance with some 

25 previously agreed standard, convention or instruction. Copies of subsequent 
layers are made in a similar manner, each with the same lateral shift as the 
previous layer or an increasing lateral shift as each layer is added. 

In order to produce the right eye image sequence a copy of layer 1 of the 
2D image is made. A copy of layer 2 is then made and placed below layer 1 with 

30 a lateral shift to the right. In the preferred embodiment the lateral shift is equal 
and opposite to that used in the left eye. For example, should layer 2 be shifted 
to the left by -2 mm then for the right eye a shift of +2 mm would be used. It 
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should be appreciated that the unit of shift measurement will relate to the medium 
the 2D image is represented in and may include, although not limited to, pixels, 
percentage of image size, percentage of screen size etc. 

A composite image is then created from the separate layers so as to form 
5 separate left and right eye images that may subsequently be viewed as a stereo 
pair. This is illustrated in Figure 3. 

In the preceding explanation it will be appreciated that the original layered 
image may be used to create one eye view as an alternative to making a copy. 

It will be understood by those skilled in the art that this technique could be 
10 applied to a sequence of images and for explanation purposes only a single 2D 
image has been illustrated. 

It will also be understood by those skilled in the art that the objects in the 
original 2D image may be described in other than visible images, for example 
vector based representations of objects. It is a specific objective of this invention 
15 that it be applicable to all image formats that are composed of layers. This 
includes, but is not limited to, cartoons, vector based images i.e. Macromedia 
Flash, MPEG encoded images (in particular MPEG 4 and MPEG 7 format 
images) and sprite based images. 
ENCODING AND COMPRESSION 
20 As we have disclosed previously there is significant redundancy in the 

allocation of depth to objects. For example, should an object appear at the same 
x,y co-ordinates and at the same depth in subsequent image frames then it is 
only necessary to record or transmit this information for the first appearance of 
the object. 

25 Those skilled in the art will be familiar with techniques to encode and 

compress redundant data of this nature. 

ALTERNATIVE EMBODIMENTS 

It will be appreciated that the lateral displacement technique can only be 

applied where objects on underlying layers are fully described. Where this is not 
30 the case, for example where the 2D image did not originally exist in layered form 

but was converted into separate layers using image segmentation techniques, 




then the previously disclosed DDC stretching techniques can be applied to create 
the stereoscopic innages. 

It will also be appreciated that by simply laterally shifting objects the 
resulting 3D image will contain objects that appear to be flat or have a "cardboard 
5 cutout" characteristic. By applying our previously disclosed DDC techniques it is 
intended that the 3D effect may be enhanced. For example, an object may have 
a depth characteristic that combines a lateral shift and a depth ramp. The 
resulting object would therefore be both laterally displaced and stretched as 
disclosed in PCT/AU96/00820. 

10 Where objects do exist in a layered from, and are partially or fully 

described, the DDC technique is not required to identify and outline objects since 
this has already been undertaken. However, the allocation of depth 
characteristics is still required. 

It will be known to those skilled in the art that stereoscopic displays are 

15 emerging that do not rely on left eye and right eye images as a basis of their 
operation. It is the intention of this invention that the techniques described may 
be employed by existing and future display technologies. 

For example, displays are emerging that require a 2D image plus an 
associated depth map. In this case the 2D image of each object may be 

20 converted into a depth map by applying the depth characteristics identifier 
previously described to each object. 

The individual layers then being superimposed to form a single image that 
represents the depth nriap for the associated 2D image, it will be appreciated by 
those skilled in the art that this process can be applied either prior to displaying 

25 the stereoscopic images or in real time. 

In addition, another display type is emerging that requires more images 
than simply a stereo pair. For example, the autostereoscopic LCD display 
manufactured by Phillips requires 7 or 9 discrete images where each adjacent 
image pairs comprise a stereo pair. It will be appreciated that the lateral 

30 displacement technique described above may also be used to create multiple 
stereo pairs suitable for such displays. For example, to create an image 
sequence suitable for an autostereoscopic display requiring 7 views the original 



2D image would be used for the central view 4 and views 1 to 3 obtained by 
successive lateral shifts to the left. Views 5 to 7 would be fornned from successive 
lateral shifts to the right. This process would be familiar to those skilled in the art 
of producing 3D images from multiple 2D images mounted under a lenticular 
lens. 

As we have previously disclosed, the depth characteristics may be 
included in the definition of the original 2D image thus creating a 2D compatible 
3D image. Given the small size of this data, 2D compatibility is obtained with 
minimal overhead. 

We have also previously disclosed that the depth characteristics can be 
included in the original 2D images or stored or transmitted separately. 

DATED this 4^' day of August 2000 
DYNAMIC DIGITAL DEPTH RESEARCH PTY LTD 
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